The field of log management and storage has developed greatly in the past 10 years, where the information that is possible to store on those systems has been consistently increasing, and the systems themselves have become faster to permit a bigger number of queries on the stored data.
These tools were originally created to assist security analysts with monitoring computer networks to pinpoint potential attacks and breaches on the networks they are defending. By putting all this data in one place, they should be able to review the totality of the log events in the network on a single environment, but that is just too much data for human analysts to make sense of.
In response to this challenge, SIEM (Security Information and Event Management) systems were put into practice, where in addition to the log management aspects described previously, deterministic correlation rule sets have been introduced. With this advancement, vast amounts of logs can be summarized in simple rule sets that trigger an event from the system if a certain type of log event happens above a certain threshold in any given time span or if an specific log event happens in sequence with another one, where those events are related do one another in some shape or form. More complicated rules can be designed by the combination and iteration of the above described, but these are the primitives that all correlation rules are based upon.
These deterministic correlation rules are able to support decision making in information security monitoring, but only to a certain extent. One of the issues is that the specific thresholds or the exact composition of rules that should be used for effective information security monitoring vary widely throughout organizational computer networks, reducing the overall utility of manual configuration of those rule sets to make sense of the ever increasing log data.
Moreover, the expression and compositions of this rules to achieve operation effectiveness in information security monitoring is very time consuming, incurring significant consulting or internal analyst costs. And even as a relative success is reached, the evolution in the organizational network assets, their normal behavior and other changes in the status quo of network configuration rapidly unbalance the work performed and invites constant review and re-tuning of those rules and thresholds.
Some advances have been attained in the implementation of behavioral and anomaly detection rules, but these have limited effectiveness as they: base their evaluation of normalcy in only a relative short period of time in the target network; and that they limit their expression in the vocabulary of the deterministic rules, trying to define thresholds and event chaining composition to express a complicated behavior that could be relevant to an analyst.
As a result of this situation, the current state of the art in using these SIEM and log management tools is plagued by an ever increasing amount of log entries, fueled by the recent advancements in computer storage and database technology. The correlation rules that were devised as tools for triage and more effective information security monitoring have been a source of noise and confusion in organizations, and the practice of information security monitoring is found in disarray. There is a need to provide a streamlined way to make informed decisions around information security monitoring activities in a day to day basis that is able to evolve with the changing network and threat landscape.